Avoiding long access latencies in redundant storage systems

ABSTRACT

Avoiding long access latencies in redundant storage systems is disclosed, including: determining a first device associated with a request is in a slow access period based at least in part on a scheduled slow access period, wherein the first device is included in a plurality of devices associated with a redundant storage system; and reconstructing data associated with the request from one or more devices included in the plurality of devices other than the first device.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 14/814,280, entitled AVOIDING LONG ACCESS LATENCIES INREDUNDANT STORAGE SYSTEMS filed Jul. 30, 2015, which is incorporatedherein by reference for all purposes, which is a continuation of U.S.patent application Ser. No. 13/421,714, now U.S. Pat. No. 9,128,910entitled AVOIDING LONG ACCESS LATENCIES IN REDUNDANT STORAGE SYSTEMSfiled Mar. 15, 2012 which is incorporated herein by reference for allpurposes, which claims priority to U.S. Provisional Patent ApplicationNo. 61/453,362 entitled AVOIDING LONG DISK ACCESS LATENCIES IN REDUNDANTSTORAGE SYSTEMS filed Mar. 16, 2011 which is incorporated herein byreference for all purposes.

BACKGROUND OF THE INVENTION

In some storage devices, as part of normal operation, the deviceoccasionally needs to perform internal activities for a period of time,such as garbage collection, defragmenting, updating metadata, etc.However, when a request to access data stored on a device is sent to thedevice while the device is performing internal activities, the accessmay sometimes take much longer than usual due to the competing internalactivities. For example, a normal response by a device to a request foraccess may take less than a millisecond. But a response by the device tothe request while the device is performing internal activities may takeup to hundreds of milliseconds.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is an example of a redundant storage system.

FIG. 2 is a diagram showing an embodiment of a RAID layer.

FIG. 3 is a flow diagram showing an embodiment of a process for avoidinglong access latency.

FIG. 4 is a flow diagram showing an embodiment of a process for avoidinga long access latency.

FIGS. 5A and 5B are diagrams showing an example of implementing process400.

FIG. 6 is a flow diagram showing an embodiment of a process for avoidinga long access latency.

FIG. 7 is a flow diagram showing an embodiment of a process forscheduling slow access periods.

FIGS. 8A and 8B are diagrams showing an example of implementing process700.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1 is an example of a redundant storage system. In FIG. 1, at leastDevices A, B, C, D, and E form a group of redundant storage devices. Forexample, in the example shown in FIG. 1, each of Devices A, B, C, D, andE may be a hard disk or a solid state drive. The Devices A, B, C, D, andE form a RAID (Redundant Array of Independent Disks) group that ismanaged in part by RAID layer 102. The Devices A, B, C, D, and E in theRAID group comprise a logical container. At least some of Devices A, B,C, D, and E are configured to store at least some data that isredundantly stored on one or more other devices of the group based on apolicy of data distribution applied to the RAID group. Typically, datastored at a RAID group is “striped” across the devices such thatlogically sequential data is segmented in a way that accesses ofsequential segments can be made to different devices. RAID layer 102 isconfigured at least in part to receive requests from an application andto send the requested data back to the application. One advantage of aRAID group is the fault tolerance provided by the redundancy of datastored among the devices of the group. In the event that a device in thegroup fails, the data stored on the failed device may be reconstructedusing data on the remaining, available devices of the group. Forexample, in a typical RAID group, parity data written on an availabledrive may be used, along with data read from other available drives, toreconstruct data that was stored on a failed device or that has becomecorrupted as stored on a device. When XOR parity is used, for example,the parity data may be used to perform XOR computations using the paritydata and related data in a set of data values that has been stripedacross devices in the RAID group to reconstruct data that cannot be readbecause a drive on which it was stored has failed. Typically, areconstruction operation is performed when a disk failure is detected,for example when a disk controller informs the RAID controller that anoperation to read data at the disk has failed.

In the example of FIG. 1, a request (e.g., a read request or a writerequest) is received at RAID layer 102. RAID layer 102 is configured tosend the received request to the devices in the group that store therequested data. Assume that in this example, the requested data isstored on at least some of Devices A, B, C, D, and E. So, RAID layer 102is configured to send the request to at least one of Devices A, B, C, D,and E that is determined to store the requested data to retrieve suchdata.

Avoiding long access latencies in redundant storage systems isdisclosed. Storage devices in a redundant storage system may beconfigured to perform internal activities during certain periods, andsuch activities may result in a device responding more slowly duringsuch periods to requests to access data stored on the device (e.g., aread request or a write request), as compared to the time required torespond to an access to the device while the device is not performingsuch internal activities. In some embodiments, a period in which astorage device performs slower or longer than usual accesses is referredto as a “slow access period.” To avoid a slow/long disk access caused bythe requested device performing internal activities or by one or moreother reasons, in some embodiments, a timeout duration is assigned toeach request associated with a first storage device such that once thetimeout duration expires for the request, the first storage device isassumed to be in a slow access period and the requested data isreconstructed using data obtained from one or more storage devices otherthan the first storage device, even if no indication has been receivedthat the operation to read the data at the first storage device hasfailed. In some embodiments, a timeout duration is dynamicallydetermined for each request based at least on one or more factors (e.g.,attributes associated with the request including the size of request).In some other embodiments, the timing of the respective slow accessperiod for each of a plurality of storage devices in a redundant storagesystem is controlled, for example by scheduling, triggering, and/orforcing the start time(s) of the slow period(s) for each storage device,such that a request that is associated with a storage device that iscurrently in a scheduled slow access period is (likely) able to beserviced using one or more other storage devices that also storeportions of the requested data and/or parity information that can beused to reconstruct the requested data and which are not (likely) atthat same time to also be in a scheduled/triggered/forced slow accessperiod.

FIG. 2 is a diagram showing an embodiment of a RAID layer. In someembodiments, RAID layer 202 is configured to manage a group of redundantstorage devices that comprise a RAID group. For example, each of thestorage devices in the group comprises a hard disk or a solid statedisk. In some embodiments, RAID layer 202 is at least part of a storagesystem that includes at least the RAID group. In some embodiments, RAIDlayer 202 and the blocks included in the layer are implemented using oneor both of software and hardware. In the example, RAID layer 202includes at least access latency avoiding block 204, data reconstructionblock 205, and RAID control block 206. The RAID control block 206 is atleast configured to receive requests (e.g., read requests or writerequests sent by an application), to service the requests, and to sendthe results (e.g., requested data) back to the requesting application.

In some embodiments, RAID layer 202 is configured to initially send arequest to one storage device in the group of redundant storage devicesbased on, for example, a determination that the storage device storesthe data that is requested and/or an identifier in the requestassociated with that storage device. In some embodiments, access latencyavoiding block 204 is configured to determine a timeout duration foreach request. In some embodiments, access latency avoiding block 204 isconfigured to determine a specific timeout duration for each requestbased on factors such as one or more of the following: a fixed timeoutlength assigned (e.g., by a system administrator) to the device, a sizeof data requested, a priority associated with the request, a priorityassociated with the device, a priority associated with the entity (e.g.,an application or other caller) that issued the request (e.g., becausesome requesting entities are more sensitive to latency than others), anda current queue length associated with the device. If the storage deviceto which the request was initially sent does not respond to the requestby the end of the timeout duration (e.g., due to the device being in aslow access period), then the storage device is assumed to be in a slowaccess period and access latency avoiding block 204 is configured toservice the request using one or more storage devices other than thestorage device to which the request was initially sent. For example,access latency avoiding block 204 can issue the request to one or moreother storage devices by invoking a reconstruction operation to beperformed by data reconstruction block 205. Data reconstruction block205 is configured to issue requests to other devices that store at leastsome of the requested data on the first device and/or parity informationthat could be used to reconstruct the data requested from the firstdevice. Once the other devices receive the reconstruction operationrequests, the other devices will respond with the relevant data and/orparity information needed by RAID layer 202 to reconstruct the requesteddata, which will be sent back to the requesting entity.

In some embodiments, access latency avoiding block 204 is configured todetermine a scheduled time for each of the storage devices in the groupto be forced to enter the slow access period. For example, a storagedevice may be “forced” to enter a slow access period using a specialcommand associated with sending/forcing the storage device into such aslow access period (e.g., using a special command that is designated bythe vendor of the storage device or by sending a command that has beenobserved or otherwise been determined to cause the device to enter itsslow access period). In some embodiments, access latency avoiding block204 is configured to determine a different scheduled slow access periodstart time for each storage device in the group of storage devices. Insome embodiments, access latency avoiding block 204 is configured todetermine the schedule of storage device forced slow access periods suchthat no two different storage devices will be in a slow access periodduring overlapping times. This way, all but one of a group of storagedevices storing at least some redundant data and/or parity informationcan still be available to service incoming requests. In someembodiments, access latency avoiding block 204 is configured to scheduleslow access periods for the storage devices in the group based on theperformances or other changing states of the storage devices. In someembodiments, access latency avoiding block 204 is configured to schedulea slow access period to occur for a storage device not substantiallymore than the frequency that the storage device would enter a slowaccess period on its own, without being forced to enter such a period.As such, when slow access periods have been scheduled for the group ofstorage devices, for a subsequently received request, access latencyavoiding block 204 is configured to determine whether the request isassociated with a storage device that is currently in a scheduled slowaccess period. In some embodiments, if the storage device is currentlyin a scheduled slow access period, then access latency avoiding block204 is configured to immediately fulfill the request using one or moreother storage devices and not wait out a timeout duration (if one hasbeen determined for the request). For example, access latency avoidingblock 204 can issue the request to one or more other storage devicesusing a reconstruction operation.

FIG. 3 is a flow diagram showing an embodiment of a process for avoidinglong access latency. In some embodiments, process 300 is implementedusing RAID layer 202.

At 302, it is determined that a first device associated with a requestis in a slow access period. In some embodiments, the device is part of aredundant storage system comprising two or more storage devices. In someembodiments, a request (e.g., a read request or a write request) isassociated with a device based on an identifier included in the requestassociated with the device. In some embodiments, the request isassociated with the device based on a determination that the devicestores the requested data. In some embodiments, the device is determinedto be in a slow access period based on a timeout duration associatedwith the request timing out. In some embodiments, the device isdetermined to be in a slow access period based on a state of the devicethat indicates that the device is currently in a forced slow accessperiod. When the device exits the slow access period the state of thedevice is updated to indicate that the device is not currently in aforced slow access period. In some embodiments, when the device isdetermined to have exited a slow access period is determined by thecompletion of the command that had forced the device into the slowaccess period.

At 304, data associated with the request is reconstructed from one ormore other devices comprising a redundant storage system in which thefirst device and the one or more other devices are included. In someembodiments, the requested data can be reconstructed with data obtainedfrom one or more devices other than the device that the request wasinitially sent to and/or is associated with. For example, the requesteddata can be retrieved using a reconstruction operation via one or moreother devices that also store portions of the requested data and/orparity information that can be used to reconstruct the requested data.In some other embodiments, instead of performing reconstruction, therequested data can be retrieved from a device in another tier ofstorage.

FIG. 4 is a flow diagram showing an embodiment of a process for avoidinga long access latency. In some embodiments, process 400 is implementedusing RAID layer 202. In some embodiments, process 300 is implementedusing process 400.

At 402, a timeout duration for a request associated with a device isdetermined. In some embodiments, for a request received at the RAIDlayer to be issued to one device of a group of redundant storage devices(e.g., SSDs), a timeout duration is determined based on one or morefactors. For example, the factors could include one or more of thefollowing: a fixed timeout length assigned (e.g., by a systemadministrator) to the device, a size of data requested, a priorityassociated with the request, a priority associated with the device, apriority associated with the entity (e.g., an application or othercaller) that issued the request (e.g., because some requesting entitiesare more sensitive to latency than others), and a current queue lengthassociated with the device. One benefit to tailoring the timeoutduration to the attributes of each request is to prevent waiting toolong to timeout a small read request, which can cause long latency, orprematurely timing out a large request, which can cause the overallstorage system to get overloaded due to extra requests. Another benefitto tuning the timeout duration to the attributes of each request is thatlonger timeout durations can be assigned to requests from requestingentities that are less sensitive to access latency, and as a result thesystem is not additionally burdened by the performance of unnecessaryreconstructions. In various embodiments, subsequent to determining atimeout duration for the request, the request is issued to the device.

At 404, it is determined whether a response from the device is receivedbefore the request times out. For example, once the request is issued tothe device, a clock will run until the determined timeout duration isover. In the event that the response is received before the timeoutduration is over, the request is considered to have been serviced, inwhich case the process ends. In the event that the response is notreceived before the timeout duration is over, the device is assumed tobe in a slow access period and control passes to 406.

At 406, data associated with the request is reconstructed from one ormore other devices. In some embodiments, the data associated with therequest is obtained via a reconstruction operation via one or moredevices in the group of devices other than the device to which therequest was originally sent/intended that also store portions of therequested data and/or parity information that could be used toreconstruct the requested data.

In some other embodiments, instead of performing reconstruction, thedata associated with the request is obtained via a request to one ormore devices at a tier of storage different than the tier of storagethat the device the request was originally sent to belongs, where thedifferent tier of storage stores at least some data redundant to datathat is stored at the other tier of storage. In various embodiments, atiered storage system includes SSD disks at the primary tier storage andhard disks at the secondary storage tiers. For example, if the requestwas originally sent to a device in a tier of storage associated withSSDs, then once the request times out, the request can be sent to one ormore devices in a tier of storage associated with hard disks.

FIGS. 5A and 5B are diagrams showing an example of implementing process400. In the examples, RAID layer 202 is implemented using the exampledescribed in FIG. 2. In FIG. 5A, a request is received by RAID layer 202and then issued to Device A. A timeout duration is determined by RAIDlayer 202 for the request. For example, the timeout duration can bedetermined prior to the issuance of the request to Device A or after theissuance of the request to Device A. Once the timeout duration has beendetermined, the clock starts to run. In FIG. 5B, it is determined thatthe request sent to Device A has timed out before Device A had anopportunity to respond (e.g., because Device A was in a slow accessperiod and/or due to one or more other reasons). Because the request hastimed out at Device A, Device A is assumed to be in a slow access periodand RAID layer 202 obtains the requested data via a techniquealternative to requesting the data from Device A. In the example, RAIDlayer 202 performs a reconstruction operation to retrieve the requesteddata from Devices B, C, D, and E, which each stores data and/or parityinformation needed to reconstruct the requested data.

FIG. 6 is a flow diagram showing an embodiment of a process for avoidinga long access latency. In some embodiments, process 600 is implementedusing RAID layer 202. In some embodiments, process 300 is implementedusing process 600.

In some embodiments, process 600 is similar to process 400 except thatmore than one request is sent to a device at a time.

At 602, a timeout duration for a first request in a queue of two or morerequests associated with a device is determined. In some embodiments, aqueue of two or more requests are associated with a device and where thedevice processes one request at a time as the other requests wait in thequeue. For example, one scenario in which a queue of two or morerequests are associated with a device is when a batch of two or morerequests are sent to the device at once. In some embodiments, for thefirst request in the queue of a device, a timeout duration is determinedbased on one or more factors. For example, the factors could include oneor more of the following: a fixed timeout length assigned (e.g., by asystem administrator) to the device, a size of data requested, apriority associated with the request, a priority associated with thedevice, a priority associated with the entity (e.g., an application orother caller) that issued the request (e.g., because some requestingentities are more sensitive to latency than others), and a current queuelength associated with the device.

At 604, it is determined whether a response from the device is receivedbefore the first request times out. For example, once the batch ofrequests is issued to the device, a clock will run until the timeoutduration determined for the first request is over. In the event that theresponse is received before the timeout duration is over, the firstrequest is considered to have been serviced and in which case theprocess ends. In the event that the response is not received before thetimeout duration is over, the device is assumed to be in a slow accessperiod and control passes to 606.

At 606, data associated with the first request is reconstructed from oneor more other devices. In some embodiments, the data associated with therequest is obtained via a reconstruction operation to one or moredevices in the group of devices other than the device to which therequest was originally sent/intended that also store portions of therequested data and/or parity information that can be used to reconstructthe requested data. In some other embodiments, the data associated withthe request is obtained via a request to one or more devices at a tierof storage different than the tier of storage to which the request wasoriginally sent/intended for, where the different tier of storage storesat least some data redundant to the data that is stored at the originaltier of storage.

At 608, data associated with each request other than the first requestin the queue is reconstructed from one or more other devices. In variousembodiments, once it is determined that the first request in the queueassociated with a device has timed out, then it is assumed that thedevice is in a slow access period and the other requests in the queueare serviced by reconstructing the data without waiting until theseother requests time out. Requests to the device that has been determinedto be in the slow access period are resumed once it is determined thatthe device is no longer in the slow access period. In some embodiments,a device is determined to be no longer in the slow access period whenone or more requests issued to the device are completed by the device(e.g., when the first request in the queue that previously timed out iscompleted by the device) or a certain period of time has elapsed sinceit was determined that the device is in a slow access mode. In someother embodiments, instead of reconstruction, the requested dataassociated with the other requests is requested from another copy of therequested data in a different tier of storage.

FIG. 7 is a flow diagram showing an embodiment of a process forscheduling slow access periods. In some embodiments, process 700 isimplemented using RAID layer 202. In some embodiments, process 300 isimplemented using process 700.

At 702, a slow access period schedule associated with at least a deviceassociated with a request is determined. In some embodiments, therequest is received at the RAID layer to be issued to one device of agroup of redundant storage devices (e.g., SSDs). In some embodiments,each of the devices in the group is scheduled to enter a forced slowaccess period based on a determined slow access schedule for the group.In some embodiments, the determined slow access schedule is configuredsuch that no two devices in the group would be in a slow access periodduring overlapping times. In some embodiments, when a device is forcedto enter a scheduled slow access period, a state of the device is setsuch that it indicates that the device is currently in a slow accessperiod. The determined slow access schedule for the storage devices canbe, for example, based on a variety of changing states in the system.For example, states can include the rates at which data is read orwritten or the history of occurrences of slow periods in the past. Insome embodiments, a device will be forced to enter into a slow accessperiod every 15 to 20 minutes depending on the rate of data beingwritten to the system.

In some embodiments, the length of time of a slow access period for adevice is usually determined to be the time between the start of theslow access period and when the device starts completing read requestsat a normal latency again. In some embodiments, the length of a slowaccess period is not fixed but dynamic and depends on factors such ashow much internal activity a device has to perform. In some embodiments,the length of time of a slow access period for a device coincides withthe length of time it takes for the command that forced the device intothe slow access period to complete. Put another way, the duration of thecompletion of the command that forced the device into the slow accessperiod coincides with the duration that the device needs to be in a slowaccess period to complete its internal maintenance.

In some embodiments, the determined schedule forces each device to entera slow access period not substantially more frequently than the deviceitself would have entered a slow access period if it were allowed toenter the slow access period on its own. So, the forcing of devices toenter slow access periods is intended to change the start times of suchperiods but not the frequencies at which they occur.

At 704, it is determined whether the device to which the request is tobe sent is currently in a scheduled slow access period. In someembodiments, a corresponding state associated with the device is checkedto determine whether the state indicates that the device is currently ina scheduled slow access state. In the event that it is determined thatthe device is not currently in a forced slow access, control passes to705. At 705, the data associated with the request is retrieved from thedevice. In the event that it is determined that the device is currentlyin a forced slow access period, control passes to 706.

At 706, data associated with the request is reconstructed from one ormore other devices. In some embodiments, the data associated with therequest is obtained via a reconstruction operation to one or moredevices in the group of devices other than the device to which therequest was originally sent/intended that also store portions of therequested data and/or parity information that can be used to reconstructthe requested data. In some other embodiments, the data associated withthe request is obtained via a request to one or more devices at adifferent tier of storage.

FIGS. 8A and 8B are diagrams showing an example of implementing process700. In the examples, RAID layer 202 is implemented using the exampledescribed in FIG. 2. In FIG. 8A, a request is received by RAID layer 202and the request is to be issued to Device A. However, upon checking astate associated with Device A that indicates whether Device A iscurrently in a forced, scheduled slow access period, it is determinedthat Device A is currently in a slow access period. For example, thestate indicating that Device A is in a slow access period can be set forDevice A when it was forced into that state using the availabledevice-specific command to do so (or it was set when one of the requeststo Device A had timed out). If the state indicates that Device A iscurrently in a slow access period, then the requested data is to beobtained elsewhere. In FIG. 8B, because it is determined that Device Ais in a slow access period, RAID layer 202 obtains the requested datavia a technique alternative to requesting the data from Device A. In theexample, RAID layer 202 performs a reconstruction operation to retrievethe requested data from Devices B, C, D, and E that each stores aportion of the requested data and/or parity information that can be usedto reconstruct the requested data.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A system, comprising: a processor configured to:determine that a first device associated with a request is in a slowaccess period based at least in part on a scheduled slow access period,wherein the first device is included in a plurality of devicesassociated with a redundant storage system; and reconstruct dataassociated with the request from one or more devices included in theplurality of devices other than the first device; and a storage deviceconfigured to store the request.
 2. The system of claim 1, wherein therequest includes an identifier associated with the first device.
 3. Thesystem of claim 1, wherein the first device is forced into the slowaccess period by an issued command associated with triggering a deviceto enter a slow access period.
 4. The system of claim 1, wherein theprocessor is further configured to determine a schedule of forced slowaccess periods corresponding to the plurality of devices, wherein theschedule of forced slow access periods indicates when each device is tobe forced into a respective slow access period.
 5. The system of claim4, wherein the schedule of forced slow access periods does not schedulefor two or more devices to be forced into slow access periods atoverlapping periods.
 6. The system of claim 4, wherein the schedule offorced slow access periods is configured such that a device does notenter a corresponding scheduled slow access period substantially moreoften than it would have if it had been allowed to enter the slow accessperiod on its own.
 7. The system of claim 1, wherein the first deviceand the one or more other devices comprise at least a portion of aredundant array of inexpensive disks (RAID) group.
 8. The system ofclaim 1, wherein to reconstruct the data associated with the requestfrom one or more devices included in the plurality of devices other thanthe first device comprises using at least parity data stored on the oneor more devices.
 9. The system of claim 1, wherein to determine that thefirst device associated with the request is in the slow access period isbased at least in part on determining that a state associated with thefirst device indicates that the first device is currently in the slowaccess period.
 10. The system of claim 9, wherein the processor isfurther configured to: determine that the first device is no longer inthe slow access period; and update the state of the first device toindicate that the first device is no longer in the slow access period.11. A method, comprising: determining that a first device associatedwith a request is in a slow access period based at least in part on ascheduled slow access period, wherein the first device is included in aplurality of devices associated with a redundant storage system; andreconstructing data associated with the request from one or more devicesincluded in the plurality of devices other than the first device. 12.The method of claim 11, wherein the request includes an identifierassociated with the first device.
 13. The method of claim 11, whereinthe first device is forced into the slow access period by an issuedcommand associated with triggering a device to enter a slow accessperiod.
 14. The method of claim 11, further comprising determining aschedule of forced slow access periods corresponding to the plurality ofdevices, wherein the schedule of forced slow access periods indicateswhen each device is to be forced into a respective slow access period.15. The method of claim 14, wherein the schedule of forced slow accessperiods does not schedule for two or more devices to be forced into slowaccess periods at overlapping periods.
 16. The method of claim 14,wherein the schedule of forced slow access periods is configured suchthat a device does not enter a corresponding scheduled slow accessperiod substantially more often than it would have if it had beenallowed to enter the slow access period on its own.
 17. The method ofclaim 11, wherein the first device and the one or more other devicescomprise at least a portion of a redundant array of inexpensive disks(RAID) group.
 18. The method of claim 11, wherein reconstructing dataassociated with the request from the one or more other devices includedin the plurality of devices other than the first device comprises usingat least parity data stored on the one or more devices.
 19. The methodof claim 11, wherein determining that the first device associated withthe request is in the slow access period is based at least in part ondetermining that a state associated with the first device indicates thatthe first device is currently in the slow access period.
 20. A computerprogram product, the computer program product being embodied in anon-transitory computer readable storage medium and comprising computerinstructions for: determining that a first device associated with arequest is in a slow access period based at least in part on a scheduledslow access period, wherein the first device is included in a pluralityof devices associated with a redundant storage system; andreconstructing data associated with the request from one or more devicesincluded in the plurality of devices other than the first device.